Memory organization in a switching device

ABSTRACT

A router for switching data packets from a source to a destination in a network in which the router includes a distributed memory. The distributed memory includes two or more memory banks. Each memory bank is used for storing uniform portions of a data packet received from a source and linking information for each data packet to allow for the extraction of the uniform portions of a data packet from distributed locations in memory in proper order after a routing determination has been made by the router.

This is a continuation-in-part of U.S. application Ser. No. 08/844,171,entitled “HIGH SPEED SWITCHING DEVICE”, filed Apr. 18, 1997, now U.S.Pat. No. 5,905,725, which is a continuation-in-part of U.S. applicationSer. No. 08/767,576, entitled “HIGH SPEED VARIABLE LENGTH BEST MATCHLOOK-UP IN A SWITCHING DEVICE”, filed on Dec. 16, 1996, now U.S. Pat.No. 5,909,440.

BACKGROUND

The present invention relates generally to data routing systems, andmore particularly to methods and apparatus for efficiently routingpackets through a network.

In packet switched communication systems, a router is a switching devicewhich receives packets containing data or control information on oneport, and based on destination information contained within the packet,routes the packet out another port to the destination (or anintermediary destination).

Conventional routers perform this switching function by evaluatingheader information contained within a first data block in the packet inorder to determine the proper output port for a particular packet.

Efficient switching of packets through the router is of paramountconcern. Referring now to FIG. 1a, a conventional router includes aplurality of input ports 2 each including an input buffer (memory) 4, aswitching device 6 and a plurality of output ports 8.

Data packets received at an input port 2 are stored at leasttemporarily, in input buffer 4 while destination information associatedwith each packet is decoded to determine the appropriate switchingthrough the switching device 6. The size of input buffer 4 is based inpart on the speed with which the destination information may be decoded.If the decoding process takes too long as compared to the rate at whichpackets are received, large sized memory elements may be required orpackets may be dropped.

In addition, the size of input buffer may be influenced by a conditionreferred to as “blocking”. Packets may be forced to remain in the inputbuffer after the destination information is decoded if the switchingdevice cannot make the connection. Blocking refers to a condition inwhich a connection cannot be made in the switch due to theunavailability of the desired output port (the port is busy, e.g.,routing another packet from a different input port). In summary, thesize of input buffer 4 is dependent on a number of factors including theline input rate, the speed of the look-up process, and the blockingcharacteristics for the switching device.

Unfortunately, conventional routers are inefficient in a number ofrespects. Each input port includes a dedicated input buffer and memorysharing between input ports is not provided for in the design. Eachinput buffer must be sized to meet the maximum throughput requirementsfor a given port. However, design trade-offs (cost) often necessitatesmaller buffers for each port. With the smaller buffers, the possibilityarises for packets to be dropped due to blocking conditions. Whileexcess memory capacity typically exists in the router (due to the variedusage of the input ports), no means for taking advantage of the excessis afforded.

To minimize the occurrence of dropping packets, designers developed nonhead-of-line blocking routers. Referring now to FIG. 1b, a conventionalnon head-of-line blocking router includes a plurality of input ports 2each including an input buffer (memory) 4, a switching device 6 and aplurality of output ports 8 each having an output buffer 9. In order toprovide non head-of-line blocking, each output port 8 is configured toinclude an output buffer 9. Each output port could simultaneously beoutputting packets as well as receiving new packets for output at alater time. As the size of the output buffer is increased, fewer packetsare dropped due to head-of line blocking at input ports.

However, these designs are even more inefficient in terms of memorycapacity and cost. Again, each output port includes a dedicated outputbuffer and memory sharing between output ports is not provided for inthe design. Each output buffer must be sized to meet the maximumthroughput requirements for a given port (in order to maintain its nonhead-of-line blocking characteristics). Even more excess memory capacitytypically exists in the router (due to the varied usage of the inputports and output ports), yet no means for taking advantage of the excessis afforded. Twice the amount and bandwidth of memory has to be usedthan required to support the amount of data being moved through thesetypes of devices.

SUMMARY OF THE INVENTION

In general, in one aspect, the invention provides a router for switchingdata packets from a source to a destination in a network. The routerincludes an input port for receiving a data packet and a physicallydistributed memory including two or more banks. Each memory bankincludes a global data area for storing portions of the data packet. Therouter further includes an input switch for streaming across the memorybanks uniform portions of the data packet, a controller for determiningpacket routing through the router, an output switch for extracting inorder the portions of packet data stored in the global data area of eachmemory bank and forwarding the packet data to an appropriate output portand an output port for transferring the data packet to the destination.

In another aspect the invention provides a router for switching datapackets from a source to a destination in a network in which the routerincludes a distributed memory. The distributed memory includes two ormore memory banks. Each memory bank is used for storing uniform portionsof a data packet received from a source and linking information for eachdata packet to allow for the extraction of the uniform portions of adata packet from distributed locations in memory in proper order after arouting determination has been made by the router.

Aspects of the invention include numerous features. The distributedmemory includes an output queue for storing a notification indicative ofthe routing of the data packet through the router. The notificationincludes linking information for retrieving at least a first cell of thedata packet from the distributed memory. The notification includeslinking information for the first 5 cells of the data packet.

The notification includes an address for an indirect cell. The indirectcell is stored in the distributed memory and includes linkinginformation for extracting cells in order from the distributed memory.

Each memory bank includes a global data area for storing portions ofdata packets and a notification area for storing notifications. Thenotification area is sized to be

{fraction (1/5)} of a size of the global data area for a given memorybank.

The router includes a plurality of multi-function multiports. Eachmulti-function multiport includes one or more input ports and outputports for receiving and transmitting data packets through the router. Aportion of the distributed memory is located within each multi-functionmultiport such that each multi-function multiport includes a memory bankhaving a global data area and a notification area. The notification areaof a given multi-function multiport stores notifications for datapackets to be routed through an output port of the given multi-functionmultiport. Memory reads and writes to and from the distributed memoryare sized to be 64 bytes.

The router includes a mapping means for mapping from a virtual addressspace to a physical address space associated with the distributedmemory. The mapping means is used for detecting aged packets in memoryand allowing for easy overwriting thereof such that garbage collectionof aged packets is not required.

In another aspect, the invention provides a method of routing a datapacket through a router in a system transmitting data packets between asource and a destination over a network including the router. The methodincludes receiving the data packet, dividing the data packet into acells of a fixed size and storing the cells in a distributed memory. Thedistributed memory includes two or more memory banks. Consecutive cellsfrom the data packet are stored in consecutive banks of the distributedmemory. Linking information is stored in one bank of the memory forlinking cells of the data packet that are stored throughout thedistributed memory. The linking information is used for extracting thecells in order for transmission from the router to the destination.

In another aspect, the invention provides a method of storing a datapacket in a router while a look-up engine determines the proper paththrough the router for the data packet in a system transmitting datapackets between a source and a destination over a network including therouter. The method includes dividing the data packet into fixed lengthcells and storing the cells and linking information for reconstructingthe data packet across a global data buffer. The global data bufferincludes two or more memory banks. The data packet is divided among thememory banks.

In another aspect the invention provides a method for determining whichdata packets stored in a router are to be passed through the router in asystem transmitting data packets between a source and a destination overa network including the router. The method includes determining a routethrough the router for a data packet. The route includes a notificationindicating a starting address in memory where the data packet is stored.The notification is processed including queuing the notification withother notifications to be output on the same output port of the routerin a stream queue. The fullness of the stream queue is determined alongwith a drop criterion based in part on fullness of the stream queue. Arandom number is generated and compared with the drop criterion. Thenotification is dropped based upon results of the comparison.

Aspects of the invention include numerous features. The fullness test isperformed when the data packet reaches a head of the stream queue. Thefullness is a percentage and the random number generated is between zeroand one.

Among the advantages of the invention are one or more of the following.A switch is provided that includes an efficient allocation of memoryacross ports and does not exhibit head-of-line blocking. The allocationenables all ports to share memory resources.

Memory bandwidth for the system may be tailored to meet specific userdefined requirements. A switch is provided in which the amount of memoryin the switch is proportional to system bandwidth where the constant ofproportionality is the round-trip network delay.

Memory bandwidth is maintained at a minimum necessary level ofapproximately twice the throughput rate. Memory bandwidth may beadvantageously expanded by the use of plug in modules.

A single system-wide virtual address space is provided that allows thememory to be read and written conveniently. An addressing scheme isprovided that maps physical memory space (and its associated availablememory modules) into a virtual space to simplify memory addresscomputations and provide a mechanism for detecting if a notification(route) is pointing to cells that have been overwritten since theoriginal data was written into memory.

Memory bandwidth for read and write operations is guaranteed for eachport.

Other advantages and features will be apparent from the followingdescription and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1a and 1 b are block diagrams of conventional router devices.

FIG. 2a is a schematic block diagram of a data routing system accordingto one embodiment of the present invention.

FIG. 2b is a schematic block diagram of a router according to oneembodiment of the present invention.

FIG. 3a is a schematic block diagram of an multi-function port accordingto one embodiment of the present invention.

FIG. 3b is a schematic block diagram indicating data transfers betweencomponents of the router of FIG. 2b according to one embodiment of thepresent invention.

FIG. 3c is a data structure for a cell transferred between amulti-function port and an input switch according to one embodiment ofthe present invention.

FIG. 4 is a schematic block diagram of a router including the timing andordering of transfers from the input port to input switch according toone embodiment of the present invention.

FIG. 5a is a schematic block diagram of an input switch according to oneembodiment of the present invention.

FIG. 5b is a schematic block diagram of a router including the timingand ordering of transfers from the input port to memory according to oneembodiment of the present invention.

FIG. 6 is a data structure for a route request stored in a key bufferaccording to one embodiment of the present invention.

FIG. 7 is a data structure for an indirect cell according to oneembodiment of the present invention.

FIG. 8 is a data structure for a cell transferred between the inputswitch and a memory bank according to one embodiment of the presentinvention.

FIG. 9 is a schematic block diagram of a reservation table according toone embodiment of the present invention.

FIG. 10 is a flow diagram of a process of loading a reservation tableaccording to one embodiment of the present invention.

FIG. 11a is a schematic block diagram of main memory according to oneembodiment of the present invention.

FIG. 11b is a schematic block diagram of a global data buffer accordingto one embodiment of the present invention.

FIG. 11c is a schematic block diagram of a router including 3 memorybanks according to one embodiment of the present invention.

FIG. 11d is a schematic block diagram of a virtual address block for usein addressing memory according to one embodiment of the presentinvention.

FIG. 11e is a schematic block diagram of an index table for use inmapping physical address space to virtual address space according to oneembodiment of the present invention.

FIG. 11f shows the relationship between physical and virtual addressesaccording to one embodiment of the present invention.

FIG. 11g is an example of a mapping between physical space and virtualspace according to one embodiment of the present invention.

FIG. 11h shows the relationship between physical and virtual addressspace over time according to one embodiment of the present invention.

FIG. 12 is a schematic block diagram of a router including the timingand ordering of transfers from the input port to memory according to oneembodiment of the present invention.

FIG. 13 is a schematic block diagram of a controller according to oneembodiment of the present invention.

FIG. 14 is a data structure for an output request transferred from thecontroller to the output switch according to one embodiment of thepresent invention.

FIG. 15a is a schematic block diagram of an output switch according toone embodiment of the present invention.

FIG. 15b is data structure for a cell output from a memory bank tooutput switch according to one embodiment of the present invention.

FIG. 16 is a data structure for a cell transferred from the outputswitch to an output port in a multi-function multiport according to oneembodiment of the present invention.

FIG. 17a is a schematic block diagram for an output section of amulti-function port according to one embodiment of the presentinvention.

FIG. 17b is a schematic block diagram for a queue system for storingnotifications according to one embodiment of the present invention.

FIG. 18 is a flow diagram for a process of routing a packet through arouter according to one embodiment of the present invention.

DETAILED DESCRIPTION

Referring to FIG. 2a, in a packet switching system, a source 10 isconnected to one or more routers 20 for transmitting packets to one ormore destinations 30. Each router includes a plurality of multi-functionmultiports that are connected to various sources and destinations. Apacket from source 10 may pass through more than one router 20 prior toarriving at its destination.

Referring to FIG. 2b, each router 20 includes an input switch 100, anoutput switch 102, a global data buffer 104 including one or more memorybanks 105, a controller 106 and a plurality of multi-function multiports150 (150-0 through 150-3), respectively. Associated with the controller106 is controller memory 109 for storing a routing table. Input switch100 and output switch 102 are connected to each multi-function multiport150 in router 20. In one embodiment, router 20 includes plug-and-playmulti-function multiports which allows for easy expansion capability.The present invention will be described with reference to a systemincluding eight multi-function multiports 150 (even though FIG. 2b onlyshows four, with each multi-function multiport including up to sixteeninput ports and sixteen output ports. Other configurations may be useddepending on user load conditions. Each multi-function multiportincludes one or more input ports, one or more output ports and a memory.The configuration and operation of the multi-function multiports will bedescribed in greater detail below.

In operation, packets are received at a multi-function multiport 150,transferred to input switch 100 and stored temporarily in global databuffer 104. When the packet is received by switch 100, a key is readfrom the first data block in the packet and transferred to controller106. The key contains destination information which is derived from theheader field associated with the first block of data in a packet andother information (such as source ID, priority data and flow ID).

A route look-up engine 110 in controller 106 performs a trie basedsearch based on the key information and returns a result which includesthe output multiport associated with the destination. The result iscoupled with other information (such as source ID, flow ID and packetlength) for routing the packet through router 20 and provided as anotification from controller 106 to output switch 102. Output switch 102transfers the notification to the identified multi-function multiport150. Upon receiving the notification information, the multi-functionmultiport 150 initiates the transfer of the packet from global databuffer 104 through output switch 102 to the appropriate multi-functionmultiport 150.

Multi-function Multiports

Referring to FIG. 3a, each multi-function multiport 150 includes aninput section 270, an output section 280 and a memory section 290.

Input section 270 includes a line input interface 300, a data handler302 and an input switch interface 304.

Output section 280 includes an output request processor 306, a lineoutput interface 308, a storage device 310, stream output buffers 312(one for each output stream), output formatter 314, an output switchinterface 316 and head and tail queue buffer 318. In addition, theoutput section includes a portion of input switch interface 304.Specifically, input switch interface 304 includes read request queues305, one for each memory bank. The use and operation of the read requestqueues, stream output buffers, and head and tail queue will be discussedin greater detail below in association with FIGS. 17a and 17 b.

Memory section 290 includes a memory bank 105 (which represents aportion of the global data buffer 104) and a notification queue body319. The use an operation of the memory section will be discussed ingreater detail below in association with FIG. 17b.

The multi-function multiport is used in conjunction with the inputswitch, output switch and controller as is shown in FIG. 3b. The variouspiece components of the input section, output section and memory sectionwill be described in greater detail below. The combination of thedevices into a single unit simplifies the interfaces between thecomponents.

Referring again to FIG. 3a, packets are received at line input interface300. As the packets are received, data handler 302 divides the packetsreceived into fixed lengths cells. In one embodiment, the length of eachcell is 80 bytes, with 16 bytes of internal header (control information)and 64 bytes of cell data. As the data handler divides the incomingpackets into fixed length cells, it synchronously outputs the cells toinput switch 100 through input switch interface 304.

The format for transfers between the multi-function multiport and theinput switch is shown in FIG. 3c. A cell 350 transferred from amulti-function multiport 150 to the input switch contains a cell header352 and cell data 354. Cell header 352 includes a type field 356, streamfield 358, and packet header fields 360. In addition, cell header 352includes an independent read request in the form of a multi-functionmultiport identifier 366 and address 368.

The type field 356 indicates the type of cell to be transferred from themulti-function multiport. At each cell slot (20 clock cycles in oneembodiment), a multi-function multiport may transfer either a data cell,an indirect cell placeholder, or a delayed indirect cell placeholder.Data cells contain data associated with an incoming packet. An indirectcell placeholder is an empty cell, and is used in conjunction withindirect addressing for the storage of the cells in the global databuffer 104. Delayed indirect cell placeholders arise when a data streamthat requires indirect addressing terminates at a time prior to thedesignated time for writing the last indirect addressing cell associatedwith the data stream to global data buffer 104. The generation andoperation of indirect placeholders and delayed indirect placeholderswill be discussed in greater detail below in conjunction with FIG. 7.

Stream field 358 indicates the stream to which the cell data belongs. Inone embodiment of the present invention, each multi-function multiportis capable of handling up to sixteen separate streams of data at a time,one on each of its respective 16 input ports.

Packet header field 360 contains header information associated with agiven packet and includes start offset information, packet length andinterface index information.

Multi-function multiport identifier 366 identifies the multi-functionmultiport which is sourcing the read request. Address 368 indicates theaddress in global data buffer 104 to be read.

Referring now to FIG. 4, a single cell 450 is transferred from amulti-function multiport 150 to input switch 100 at each cell (time)slot “T”. For a given cell slot “T”, input switch 100 receives a totalof “N” cells, where “N” is equal to the number of multi-functionmultiports.

In one embodiment, cells from a given stream may be written to memory inan order that is different from the arrival order. These out of orderwrites are performed to make efficient use of scarce bandwidth betweenthe multi-function multiports and the input switch. When a packet comesin to the multi-function multiport, it is broken up into cells as thebytes arrive and the cells are placed in per-bank output queues on theway to the input switch. These queues are designed to share scarceinterconnect bandwidth between the streams of a multi-functionalmultiport in the most efficient way possible, but they have thedetrimental effect of reordering cells at the interface between themulti-function multiport and the input switch. Thus the cells from agiven stream may arrive at the input switch out of order. Themulti-function multiport marks the data cells of a stream with one offour codes: first cell (FC); intermediate data cell (DC); last cell(LC); or first cell which happens to be also a last cell (FLC).

Input Switch

Referring to FIGS. 2b and 5 a, input switch 100 includes a round robindata handler 500, one or more input interfaces (501-0 through 501-7, onefor each multi-function multiport 150), one or more memory interfaces502 (502-0 through 502-7, one associated with each memory bank), a likeplurality of pointers 504 (504-0 through 504-7), an output processor505, one or more output interfaces 506 (506-0 through 506-7, one foreach multi-function multiport 150), a reservation table 508, an indirectcell processor 510, controller interface 512 and read controller 517.

a) Transfers through the Input Switch

Round robin data handler 500 receives cells from each multi-functionmultiport and transfers them to output processor 505 for output to anappropriate memory bank 105 in global data buffer 104. Round robin datahandler 500 services the inputs (cells) received on input interfaces 501in a round robin, time division multiplexed manner. That is, for a givencell slot, one cell from each multi-function multiport is received atthe round robin data handler 500 and subsequently transferred to outputprocessor 505 for transfer at the next cell slot to a memory bank 105 inglobal data buffer 104. At the next time cell slot, data handler 500transfers the next cell received from the same multi-function multiportto output processor 505 for transfer to a different memory bank. In oneembodiment, the next cell received is transferred to the next memorybank (next in numerical order, modulo N) in the memory array.Alternatively, another time dependent permutation may be used to controlthe transfer of successive cells from the same multi-function multiport.

Referring to FIG. 5b, the timing and ordering of transfers from themulti-function multiport to memory is shown. For the purposes of thisexample, a sequence of cells is depicted on each transmission line. Forthe purposes of this example only, each transmission line is consideredto be very long and contains data associated with two or more cells. Inoperation, the transmission lines are short and multiple cells are notpresent on a transmission line at a given time. At cell slot T4 a seriesof cells 450-0 through 450-7 are transferred down transmission lines458, one from each multi-function multiport 150 to input switch 100. Atcell slot T3 (one just prior in time to cell slot T4) a series of cells452-0 through 452-7 are transferred down transmission lines 458, onefrom each multi-function multiport 150 to input switch 100.

Round robin data handler 500 and output processor 505 within the inputswitch 100 transfer cells out to global data buffer 104 on transmissionlines 460. As can be seen at cell slot T2, output processor 505 outputsone cell 454-B₀ to 454-B₇ to each memory bank in a single cell slot. The“B_(x)” designator indicates the multi-function multiport from which theparticular cell was received. One cell from each multifunction multiportis written to global data buffer 104 per cell slot. At time period T1(one cell slot prior to cell slot T2), again one cell (456-B₀ 456-B₇) iswritten to each memory bank. Round robin data handler 500 time divisionmultiplexes the transfers to output processor 505 such that consecutivecells from the same multi-function multiport are written to consecutivememory banks 105 (modulo N) in global data buffer 104.

Referring again to FIG. 5a, pointer 504 indicates the location in anassociated memory bank to which the next cell will be written. Outputprocessor 505 writes a cell to a memory location in a particular memorybank based on the next available address in the bank as is indicated bythe associated pointer 504.

b) Key Reading and the Linking Process

Round robin data handler 500 includes a key reading engine 514 fordetermining the key information associated with a first cell in a packetand a linking engine 515 for linking cells in the same packet.

The process of reading key information is known in the art. After thekey is determined for a given packet, it is stored temporarily in keybuffer 516 in input switch 100 until the entire packet has been storedin global data buffer 104. The data structure for entries 600 in the keybuffer 516 is shown in FIG. 6. Each entry or “info cell” 600 includes akey 602, full address 604, offsets 606 and an indirect cell indicator608.

Referring now to FIGS. 5a, 6 and 7, linking engine 515 determines thestarting address (full address) in memory for where the first cell in agiven packet is to be stored in memory. The starting address includesthe bank number in global data buffer 104 (the bank number which isassigned to store the cell by round robin data handler 500) and thefirst available address location in the designated bank (as is indicatedby the associated pointer 504). The starting address (full address 604)is stored in key buffer 516 along with the associated key 602 for thepacket. When the next cell associated with the same packet arrives atswitch 100, an offset 606 associated with the offset at which the cellis to be written (relative to the full address) is computed and storedin key buffer 516. In one embodiment of the present invention, up tofour offsets 606 are stored. Each offset address is computed based onthe relative offset in memory between the location of the last cell inmemory and the value of the pointer 504 associated with the currentmemory bank which is to be written.

If more than five data cells are included in a packet, then the indirectcell indicator for that packet is set, and the last offset indicates theaddress in memory where the first indirect cell associated with thepacket is stored. Indirect cells will be described in greater detailbelow in reference to FIG. 7. After the packet has been stored inmemory, the associated info cell in key buffer 516 (a route look-uprequest) is forwarded through the controller interface 512 to thecontroller 106 for processing. Alternatively, the info cell may betransferred after the first five cells have been stored in memory.

The linking or threading of cells for a packet is performed by using theoffsets described above and indirect cells. Offsets are used to linkcells in a packet. Offsets may be stored along with key information androuted through controller 106 (FIG. 2b) or may be stored in indirectcells. In one embodiment, if a cell contains 5 cells or less, noindirect cells are required to be used. Indirect cell processor 510performs the linking of cells in memory for a given packet. Indirectcell processor 510 generates indirect cells for storage in global databuffer 104. Indirect cells contain offset information associated withthe relative offset in memory space between contiguous cells in thepacket. Indirect cell processor includes indirect cell memory 520 forstoring indirect cell data during the formation of indirect cells.

Referring now to FIG. 7, the data structure for an indirect cell 700includes a linking field 702, a plurality of offset fields 704, and alast field 706. Linking field 702, when not set, indicates the currentindirect cell is the last cell in the chain of indirect cells for agiven packet. If set, then more indirect cells exist for the givenpacket. If more indirect cells exist, then last field 706 indicates theoffset to the location in memory of the next indirect cell associatedwith the packet. In one embodiment, indirect cells contains up to 56offset data blocks for linking 56 cells in memory.

As was described above, when a packet is received, the linking engineprocesses the first five cells and stores linking information in theform of a start address and four offsets in key buffer 516. In the eventmore than five cells are contained within a packet, the indirect cellprocessor takes over for the linking engine and computes the offsetsassociated with the locations in memory where the remaining cells in thepacket are stored. Round robin processor 500 passes cells to the outputprocessor 505 for transfer to an associated memory bank in global databuffer 104. Round robin processor 500 enables the indirect cellprocessor when the packet being processed contains more than 5 cells(based on header information included within the first cell). At thetime for writing the fifth cell to memory, indirect cell processor 510stores in indirect cell memory 520 the address (the “indirect celladdress”) associated with the location in memory at which the fifth cellwould have been written if it had been the last cell in the packet. Theindirect cell address indicates the location in memory where theindirect cell is to be written when full (or when the last cell of thepacket is processed).

When an indirect cell is full (having stored offsets in all availablelocations except the last field 706), then the indirect cell processorstores the offset associated with the location in memory where the nextindirect cell is located in the last field 706. Thereafter, the fullindirect cell is written to its appropriate place in memory. The writingof the indirect cell to memory coincides with the receipt of an indirectcell placeholder by the input switch 100 from the associatedmulti-function multiport 150. This process continues until the last cellin a packet is stored in memory. At that time, the last indirect cell iswritten to memory, and the associated entry 600 from the key buffer 516is transferred to the controller 106 for processing. For a given packet,all indirect cells are written to the same memory bank in the globalmemory buffer.

As often will be the case, the last cell of a packet will not coincidewith the timing required to write the completed indirect cellimmediately into memory. This is because packet length is completelyarbitrary. The end of a packet will likely not coincide with the lastavailable entry of an indirect cell. When a packet has completed (allcells have been received by the input switch) and a last entry in theindirect cell is written, the indirect cell is free to be written tomemory. However, the writing will be delayed until the proper time,hence the term delayed indirect cell. A delayed indirect cell is aindirect cell that is the last indirect cell associated with a packet.It is delayed, because it is written to memory after the rest of thepacket has been written to memory. The timing of the write to memory isdictated by the address which is reserved for the indirect cell. As wasdescribed above, at the time for the creation of an indirect cell, itsposition in memory is reserved. The delayed indirect cell will bewritten to memory at the next time slot available for the particularmulti-function multiport to write to the particular memory bank afterthe packet has been completed. The timing of the write to memory ofdelayed indirect cells coincides with the receipt of a delayed indirectplaceholder from the appropriate multi-function multiport 150.

c) Transfers to Memory

The data structure of a cell transferred from input switch 100 (via theoutput processor 505) to a memory bank 105 in global data buffer 104 isshown in FIG. 8. The unit of addressing and memory allocation is a64-byte cell, and all accesses to memory are either cell reads or cellwrites. A cell size of 64 bytes was chosen as a compromise between theconflicting requirements of bandwidth efficiency and storage efficiency.DRAM bandwidth efficiency dictates larger sizes, while storage losscaused by internal fragmentation when fitting variable size packets intofixed size cells dictates smaller sizes.

At each cell slot, output processor 505 generates a cell 800 whichincludes a read request source field 802, read address 804, writeaddress 806 and data field (cell data received from multiport 150) 808.The read request source field 802 indicates the output port (in theparticular multi-function multiport 150) requesting the read(destination output port). Output processor 505 receives read requestsfrom read controller 517 and bundles the read request with any writerequest received from round robin data handler 500 destined for the samememory bank. At each cell slot, output processor 505 provides a cell 800which may include a write and read request to each memory bank 105 inglobal data buffer 104.

Read controller 517 controls the transfer of read request signalsflowing from input switch 100 out memory interface 502 to the individualmemory banks in global data buffer 104. Read controller 517 receivesread requests from each multi-function multiport through outputinterfaces 506. The format of each request includes sourceidentification (output port) and a full address in memory which is to beread. At each cell slot, each multifunction multiport port may generatea read request for processing by switch 100 to read a memory location inglobal data buffer 104, resulting in the reading of a cell (a readreply) from a memory bank 105 (on a subsequent cell slot) to outputswitch 102.

Read controller 517 loads a reservation table 508 as requests totransfer packets are received from the various multi-function multiports150. The reservation table is loaded such that at every cell slot asingle read request is generated for each bank of memory 105. Referringnow to FIG. 9, reservation table 508 includes a plurality of columns900, one for each memory bank 105 in global data buffer 104, a pluralityof rows 902, placeholders 904 and loaded entries 906. Each rowrepresents a set of read requests (one per memory bank) to be generatedon a single cell slot. Each row includes a single entry for eachmulti-function multiport 150. At each cell slot, each multi-functionmultiport is capable of requesting a read from a single memory bank 105in global data buffer 104. Associated with reservation table 508 is aread pointer 908. The pointer points to the next row in the reservationtable to be read. Rows ahead of the read pointer correspond to requeststhat will be queued at a later cell slot time. In one embodiment, thepointer moves at least one row in each cell slot time.

Loaded entries 906 reflect read requests to be performed as a result ofreservation requests received from individual multi-function multiports.Placeholders 904 represent available slots in which read requests for aparticular memory bank are still available (e.g., read requests whichhave not as of yet been received for this memory bank from a particularmulti-function multiport). At each cell slot, the read controller 517performs three functions: loading entries in the reservation table atthe first available location in the table (after the read pointer),outputting the last row as read requests to the output processor 505;and refreshing the table, moving out the last row, incrementing the rowsand creating a new row at the top of the table. The number of rows inthe reservation table must be as large as the product of the latency inprocessing read requests multiplied by the number of banks. In oneembodiment, 48 rows are included in reservation table 508 reflecting asystem including six cell slots of latency and eight memory banks.

At initialization, reservation table 508 contains placeholders 904 inall of the rows 902. Placeholders 904 are locations in the reservationtable which have not been loaded. As read requests are processed by theread processor, certain ones of the placeholders 904 are converted toloaded entries 906 based on the read requests. Loaded entries 906include a read request address.

Referring now to FIG. 10, the process of loading the reservation tableincludes receiving a read request (full address) from an multi-functionmultiport (1000). The read controller decodes the read request todetermine the column (based on the memory bank to be read from) in thereservation table to search (1002). The read processor searches,starting at the location indicated by pointer 908 in the reservationtable, for the first placeholder associated with the multi-functionmultiport that generated the read request (1004). The read processortransforms the placeholder 904 to a loaded entry 906 by writing the fulladdress of the read request at the location (1006). The process repeatsfor each read request received by the read controller (1008).

Memory Architecture

Referring now to FIGS. 11a-c, main memory 1050 is used as temporarybuffer storage for packets flowing into the system on input streams 1052and out of the system on output streams 1054. Main memory is dividedinto two distinct parts: a global data buffer 104 that is used to storeincoming packets while the lookup engine determines the outgoing streamfor each packet; and packet notification queues 319 that are used tostore packet pointers (notifications) after the outgoing stream has beendetermined. Notification queues 319 are associated with outgoingstreams, whereas the global data buffer 104 forms a common pool sharedamongst all the streams.

Global data buffer 104 includes a plurality of memory banks 105.Associated with each memory bank is an input switch interface (an inputport) 304 and output switch interface (an output port) 316. At each cellslot, each memory bank receives at most one write and one read requestvia input switch interface 304. The write requests are associated withcells received from a multi-function multiport 150. Read requestsreflect a request for cell data to be transferred from a memory bank 105to output switch 102 for ultimate transfer to a requestingmulti-function multiport 150.

The memory in the multi-function multiport configuration is physicallydistributed across a number of banks b, one bank for each activemulti-function multiport in the system. FIG. 11c show a system includingthree banks, numbered 0, 1 and 4 respectively, representative of threeactive multi-function multiports. Each bank is divided into twocontiguous, non-overlapping regions referred to as global data area(memory bank 105) and the notification area (notification queue 319).The global data area for a bank constitutes 1/b of the memory of theglobal data buffer 104. The notification area provides space for queuingnotifications that will be sent out on the line output interface 308 fora given multi-function multiport. Typically, the global data area isfour times larger than the notification area; this factor derives fromthe ratio between data size and notification size for the shortestpacket.

Each bank's memory bandwidth is sufficient for reading and writingpackets from a full-duplex OC-48 interface as well as for queuing anddequeuing notifications for the worst-case example of single-cellpackets. Thus, both the aggregate memory size and the aggregate memorybandwidth scale linearly with the number of active multi-functionmultiports b in the system.

In one embodiment, each memory bank is implemented as two sub-banksusing two 72-bit wide SDRAM (static dynamic random access memory) DIMM's(dynamic in-line memory modules) cycling at 125 MHZ . The sub-banks aretransparent to the input and output switch resulting in what appears tobe one continuous bank from the perspective of the switches. However,the sub-bank architecture allows for better throughput. Each DIMM has a72-bit wide ECC (error correction code) protected data path going to 9SDRAM chips each of which is 8 bits wide. The two DIMM's have separateaddress busses and are addressed independently of one another. TheDIMM's are interleaved on bit 0 of the 23-bit address. In oneembodiment, the smallest memory bank configuration is 32 MBytes, using16 Mbit chips and the largest is 512 MBytes, using 256 Mbit chips.

As was described above, a bank can receive at most one read request andone write request every cell slot. Since a cell slot is 20 clock cyclesat 125 MHZ , this works out to a peak bandwidth demand of 400 MBytes/secfor reads and 400 MBytes/sec for writes. The worst case notificationload occurs for single cell packets. For unicast traffic, this load isexactly ¼ the data bandwidth which works out to 100 MBytes/sec for readsand 100 MBytes/sec for writes. In this embodiment, the total peak memorybandwidth needed is therefore 1 GByte/sec.

In this embodiment, the peak transfer rate of each DIMM is 1 GByte/sec,but the sustained rate depends on the actual mix of reads and writes andhow the addresses are distributed over the internal DIMM banks. Inpractice, each DIMM is expected to deliver a sustained data rate ofaround 650 MBytes/sec. The total of 1.3 GBytes/sec supplied by the twogroups is 30% larger than the maximum sustained requirement of 1GByte/sec/. The 30% headroom provides a way to sustain instantaneousloads where one DIMM has more references directed to it than the other.The memory controller for the two DIMM's resides in the multi-functionmultiport.

In one embodiment, all banks are made the same size and approximately⅕th of the memory in each bank is allocated to the notification area and⅘th to the global data area. The purpose of this allocation is to makeit exceedingly unlikely for a stream to run out of memory because ofspace in its notification queue. With a worst case packet size of 64bytes, notifications (sized at 16 bytes) need {fraction (1/4)} th theamount of storage that packet data needs, which is exactly theproportion allocated. Any cell in the global data buffer may be accessedvia its physical cell pointer, which identifies the physical bank numberand the address of the cell within the bank. The physical cell pointerdefines a system-wide physical address space. To simplify addresscomputations, as well as to provide a mechanism to detect old packets,accesses to the global packet buffer are performed through a system-widevirtual address space that maps to the physical address space.

Incoming packets are broken up into as many cells as needed and thecells are written to the global packet data buffer as they arrive asdescribed above. The global data buffer is treated as a single largecircular buffer. The input switch maintains an array of write pointers,one per active bank, to keep track of where to write the next cell. Thepointers start out at one end of the buffer and advance until theyeventually wrap around and overwrite packets that were written a longtime ago. An ageing mechanism is used to guard against reading cellsthat may have been overwritten by subsequent packets. The cells ofpackets arriving on a given stream are interleaved strictly across theactive banks to spread the bandwidth load.

a) Addressing

Global data buffer 104 is divided into banks which are in turn dividedinto fixed size, non-overlapping 64-byte units called cells. The cell isboth the unit of addressing and of memory allocation. All accesses tomemory are performed using two operations: cell read and cell write.Since there is only one access size, memory allocation isstraightforward and there are no issues of data alignment or byte order.

Each cell in the system is uniquely identified by a physical cellpointer or PCP, which identifies the cell's physical bank number andaddress within that bank. The set of PCP's form a system-wide physicaladdress space that can be used to read or write any cell in the system.While this space provides access to all possible banks in main memory,it is not convenient for reading and writing cells that are part of theglobal data buffer.

For example, physical space may have holes in it corresponding to memorybanks (multi-function multiports) that are either not plugged in or areplugged in but inactive.

In addition, the size of the global data area is not necessarily a powerof two (since an odd number of multi-function multiports may beavailable and because of the ⅘ to ⅕ ratios set forth for the global areaand notification areas respectively), so address computations involvechecks that would normally not be needed. Both these factors make theaddress arithmetic that needs to be performed on each access awkward,and using physical space directly forces each entity issuing reads andwrites to know how to do this arithmetic. Finally, direct access tophysical space provides no mechanism for detecting if a notification ispointing to cells that have been overwritten since the original data waswritten to memory. A virtual-to-real address mapping mechanism providesa way to solve these problems. This mechanism defines a singlesystem-wide virtual address space that allows the global packet bufferto be read and written conveniently. All non-diagnostic accesses to thebuffer use this address space.

b) Physical Space

Each memory bank has a unique 3-bit physical bank number, or PBN, thatis equal to the number of the slot in which the bank is plugged. In oneembodiment, slots are numbered sequentially from 0 to 7 (assuming aneight multi-function multiport configuration). Note that a bank may beplugged into any available slot so that the set of active bank numbersin a system can be an arbitrary subset of the numbers [0 . . . 7]. ThePBN serves as the physical address of a bank.

Within a bank, cells are identified using a 23-bit physical celladdress, or PCA (1124 of FIG. 11f). Cells are numbered sequentiallystarting at 0 and ending at ^(N) _(bank) ⁻¹, where ^(N) _(bank) is thenumber of cells in the bank. In one embodiment, the maximum amount ofmemory in a bank is limited by the architecture to 2²³ cells, or 512MBytes.

A bank's memory is divided into two contiguous non-overlapping regions:cells [0. . . ^(N) _(buf−1)] are allocated to the global packet buffer,while cells [^(N) _(buf) . . . ^(N) _(bank)] are allocated to queuesthat hold notifications for packets to be sent out via the associatedmulti-function multiport. The number of cells N_(buf) allocated to theglobal data buffer is the same for each of the b active banks so eachbank contributes exactly 1/b^(th) of the storage for the buffer.Typically, around ⅕th of the memory in a bank is allocated fornotification queues and ⅘th for the global data buffer.

Although each bank is shown to be the same size, note that only theglobal packet buffer areas are required to be the same size acrossbanks. The architecture places no such restriction on the size of thenotification areas even though they will typically be the same size.

Any cell in the system is identified uniquely by the 26-bit tuple<PCA><PBN>. This tuple is called the cell's physical cell pointer, orPCP. The set of possible PCP values defines a system-wide physicaladdress space. All cell references must be translated to a PCP beforethe cell can be read or written. In one embodiment, the maximum amountof memory in the system is limited by the architecture to 2²⁶ cells, or4 GBytes.

c) Virtual Space

Referring now to FIG. 11d, there is a single, system-wide virtualaddress space that allows the global data buffer to be read and writtenconveniently and also allows for the easy detection of aged packets inmemory. By providing a mapping from virtual to physical address spacemaintenance of memory is simplified and no garbage collection of agedpackets is required. The virtual space is defined by the set of possiblevalues for a 27-bit quantity called the virtual cell pointer, or VCP(1120). The VCP consists of a 24-bit virtual cell address, or VCA(1122), concatenated with a 3-bit virtual bank number, or VBN as shownin the figure 11c.

The set of values for VBN define a virtual bank number space that makeit convenient to do arithmetic on bank numbers. If there are b activebanks in the system, the legal values of VBN are [0 . . . b−1], and allarithmetic on VBN's is done mod b.

The set of values for VCA define a virtual cell address space for thebank identified by VBN. This virtual cell address space makes itconvenient to do arithmetic on cell addresses that lie in its bank andalso enable ageing checks.

A VCP is translated to a physical cell pointer (PCP), by performing twoseparate mappings: first, the VBN is mapped to a physical bank number(PBN) and then the VCA is mapped to a physical cell address (PCA). Thetwo mappings are not independent. In particular, the VCA to PCA mappinguses states particular to the physical bank produced by the firstmapping.

d) Bank Number Mapping

Virtual bank numbers are translated to physical bank numbers by theinput switch using an 8-entry lookup table 1150 as shown in FIG. 11e.The table is indexed by a 3-bit VBN and produces a 3-bit PBN as theresult. The table The example shows the table for a system that has fiveactive multi-function multiports plugged into slots 2, 3, 4, 6, and 7.

In general, if there are b active banks in the system, the first bentries of the table are filled with the physical bank numbers for thesebanks.

e) Cell Address Mapping

Referring now to FIG. 11f, virtual cell addresses are also translated tophysical cell addresses by the input switch. As mentioned earlier, VCAto PCA translation is always done in the context of a particularphysical bank which, by definition, must be active.

For purposes of simplifying mapping, the virtual and physical celladdress spaces are broken up into contiguous, self-aligned, 4096 cellregions called pages. The low-order 12 bits of both VPA's and VCA'scorrespond to the address of a cell within a page, while the high-orderbits correspond to the address of the page itself. Referring now to FIG.lie, the mapping process preserves the low-order bits of the addresswhile mapping the high order bits.

The operation of the VP_(Map) (1130) function is as follows: Let N bethe number of pages contributed to the global packet data buffer by eachbank. From the structure of VCA's, it is clear that this number must bean integer in the range [1 . . . 2048] (The value 0 is excluded becauseit represents the case of no buffer memory). Note that the number ofpages in virtual space is exactly two times the largest value of N. Aswill be seen later, this fact is crucial for the ageing scheme to work.

The VP_(Map) function is time-dependent, but its operation at a giventime is straightforward. For each bank this function maps a contiguousregion of N pages in virtual space [V_(TP), V_(TP)+1, . . . , V_(LP)]onto the N pages of physical space for that bank (where TP stands for“trailing page” and LP for “leading page”). All regions are located atthe same point in their respective spaces at a given time. For thisdiscussion, all arithmetic is mod 4096, so virtual address space isassumed to wrap around after page 4095. One of the virtual pages in theregion is always mapped to physical page 0. Let this page bev₀=v_(TP)+k. Then, virtual pages [v_(TP)+k, v_(TP)+k+1, . . . , V_(LP)]are mapped directly to physical pages [0, 1, . . . , N−k−1], and virtualpages [V_(TP), V_(TP)+1, . . . , V_(LP)+k−1] are mapped directly tophysical pages [N−k−1, N−k, . . . , N−1]. That is, the N page region isbroken into two disjoint pieces and pages in the two pieces are mappedone-for-one to identically sized pieces of physical space. References tovirtual pages that lie outside the region are mapped to physical page0×FFF to indicate that the reference was to a cell that has beenoverwritten. FIG. 11g shows this mapping for a simple example where N=9,and k=3.

The time-dependent behavior of VP_(MAP) is that the N page region ofvirtual space that is mapped moves with time. At system initializationV_(LP) is set to 0, and V_(TP)is set to −(N−1). As cells are written tosystem memory, the write pointers for each bank advance more or lesstogether but they do not move in lock-step. The system computes V_(LP)as the high-order page bits of the bank write pointer that is furthestahead; V_(TP) then is simply (V_(LP)−N+1). FIG. 11h shows how the regionmoves with time. The virtual address space is shown as a circular bufferwith page numbers marked on the outside, and the mapped region is shownhighlighted. The leftmost picture shows the mapping at initializationtime, while the next two show the mapping at progressively later times.

As the region advances, the value of v₀ must be updated periodically soit stays in the range of virtual addresses that is being mapped. Atsystem initialization v₀ is set to 0, and subsequently each time v₀becomes equal to V_(TP), it is incremented by N. The increment is doneafter mapping, not before.

reset: v₀=v_(LP)=0;

if

(v₀=v_(TP))

v₀=v₀+N

f) Transfers from the Input Switch to Memory

Referring now to FIG. 12, the transfer of cells from the input switch100 to global data buffer 104 is performed in a time division multiplexfashion. That is, consecutive cells from a given multiport are directedto different memory destination locations. At each time period (cellslot), the input switch transfers to memory a single cell received fromeach multi-function multiport (as available) into memory. At a next timeT+1 the input switch transfers again a single cell from each multiportinto memory. Successive entries from the same input multiport arewritten to different memory banks 105 in global data buffer 104.

Controller

Referring now to FIG. 13, controller 106 includes controller memory 109,route look-up engine 110, input switch interface 1300, and output switchinterface 1302. Controller 106 receives a route look-up request frominput switch 100 at the input switch interface 1300. In one embodimentof the present invention a plurality of route look-up engines 110 areincluded in controller 106, each receiving look-up requests inround-robin fashion so as to speed the routing process. In oneembodiment, controller memory 109 is a four-bank static random accessmemory (SRAM) that requires thirty two route look-up engines 110 toservice at full bandwidth. The matching of keys retrieved from a packetin order to determine a best match route through the router is describedin greater detail in co-pending patent application entitled “HIGH SPEEDVARIABLE LENGTH BEST MATCH LOOK-UP IN A SWITCHING DEVICE”, filed on Dec.16, 1996, by Fergusen et al., Ser. No. 08/767,576, which is herebyexpressly incorporated by reference.

The route look-up engine servicing the route look-up request performs abest match look-up and outputs a notification through output switchinterface 1302 to output switch 102. The notification includes a resultwhich indicates the multi-function multiport to be used in the transferof the packet to its destination.

Referring now to FIG. 14, the data structure associated with thenotification outputted by the controller 106 to the output switch 102 isshown. Th data structure 1400 for the notification includes a mask 1402,a next hop index pointer 1404, full address 1406, offsets 1408 andpacket length 1410.

The mask field 1402 is used to indicate which multi-function multiportconnected to output switch 102 is to transfer the packet. In oneembodiment, the notification may be sent to more than one multi-functionmultiport resulting in the broadcast of the associated packet.

Associated with each multi-function multiport 150 is a storage 310. Thenext hop index pointer points to a location in storage 310. Storage 310is used to store media header information associated with a particulartype of packet transfer. Next hop addresses, media headers and storage310 will be described in greater detail below in association with theoutput section of multi-function multiport 150.

The full address 1406 indicates the starting address in the global databuffer where the first cell in the packet is stored. As was describedabove, offsets 1408 provide linking information for retrieving cells oran indirect cell associated with the packet. Packet length filed 1410indicates the length of the associated packet and may be used todetermine if indirect cells will have to be retrieved.

Output Switch

Referring now to FIG. 15a, output switch 102 includes a controllerinterface 1500, one or more memory inputs 1502 (1502-0 through 1502-7,one for each memory bank), one or more outputs 1504 (1504-0 through1504-7, one for each multi-function multiport), a result processor 1506and an output processor 1508. Output switch 102 performs four functions:receive output results, process output results, receive cells frommemory and output cells to output ports.

a) Transfers from Memory to the Output Switch

Cells from memory are received at memory inputs 1502 and transferred tooutput processor 1508. Cells are transferred based on read requestsreceived at the input switch from multi-function multiports. The datastructure associated with the cells transferred from global data buffer104 to output switch 102 is shown in FIG. 15b. Each cell 1510 includesan output port identifier 1512 and cell data 1514.

Output processor 1508 decodes the destination multi-function multiportfrom the cell information received from memory and transfers the celldata to the appropriate outputs 1502. At each cell slot, output switch102 may receive a cell for processing from each bank in global databuffer 104.

b) Transfers from the Output Switch to the Multi-function Multiports

Output switch 102 receives notification from controller 106 oncontroller interface 1500. Result processor 1506 decodes the result(route) and determines which multi-function multiport(s) 150 is (are) toreceive the route data. Based on mask 1402 in the notification, resultprocessor 1506 transfers the notification to output processor 1508 fortransfer to each multi-function multiport 150 indicated. At each cellslot, output processor 1508 provides (via outputs 1504) a route to eachmulti-function multiport 150.

The data structure associated with the data transferred from outputprocessor 1508 to multi-function multiports 150 is shown in FIG. 16. Acell 1600 includes a header 1602 and data field 1604. The header 1602includes memory bank source information 1606 and route information 1608.The memory bank source information includes a source identifier forindicating which memory bank provided the cell in data field 1604. Routeinformation 1608 contains data from the notification including a nexthop index, packet length, full address and offsets.

Output Section of a Multi-function Multiport

Referring now to FIGS. 17a and 17 b each multi-function multiport 150includes an output switch interface 316, an input switch interface 304including read request queues 305, head and tail queue buffer 318, anoutput request processor 306, an line output interface 308, storagedevice (memory) 310, stream output buffers 312 and output formatter 314.

a) Notification Queues

A multi-function multiport 150 receives notification that a packet is tobe processed in the form of a notification cell 1600 (FIG. 16) receivedat the output switch interface 316.

Output request processor 306 processes notifications, storing each in anappropriate location in head and tail queue buffer 318 and servicingnotification requests as they make their way through the variouspriority queues in head and tail queue buffer 318. The servicing ofrequests results in the generation of a read request to input switch 100associated with the first address in memory where the packet (associatedwith the particular notification) is stored.

Referring now to FIG. 17b, head and tail queue buffer 319 includes aplurality of notification queues Q 1700, where Q=4*s, and where s is thenumber of active streams in the multi-function multiport. Unlike theglobal data buffer, the queues Q are implemented on a per-port basis.The queues in a respective multi-function multiport store only thosenotifications associated with streams to be outputted from therespective port of the multi-function multiport. Each queue is itselfdivided into a head region 1702, a tail region 1704 and a body region1706. The head and tail region for a queue are stored in the head andtail queue buffer 318.

The size of the portion of the head and tail queue buffer dedicated toeach stream is fixed at initialization time and is proportional to thepeak bandwidth of its stream. The partitions between portions are “hard”in the sense that a stream cannot use more than the memory allocated toit. The partitions between queues associated with the same stream are“soft”. The size of an individual queue is proportional to the nominalbandwidth allocated to its queue. The body region of the notificationqueue is stored in the notification area 319 (FIG. 3a) of the memorysection 290 of the multi-function multiport 150. Each stream is assigned4 queues (the body portions of the priority queues) in the notificationarea 319 (FIG. 3a). The body region is sized to be ⅕ of the overallmemory section.

Each queue associated with a given stream is serviced according to apriority scheme. Notifications that are received by the output requestprocessor 306 are loaded into an appropriate queue associated with astream based on the priority of the notification. Priority fornotifications can be set by an external source and may be included inthe packet received by the router. Alternatively, controller 106 (FIG.3a) may set the priority depending on the amount of time required toperform the route look-up or other criteria.

Once a queue 1700 has been identified based on the priority informationand stream ID, the output request processor 306 loads the notificationinto the appropriate tail queue 1704. Notifications are transferredbetween the respective head, tail and body portions of a queue based onavailable space by a queue manager (not shown). In one embodiment, eachnotification is 16 bytes, and the notification area 319 is sized to hold64 bytes. Accordingly, for reasons of bandwidth efficiency, all readsand writes to the notification area are done using 64-byte cellscontaining four 16-byte notifications each.

The head and tail of each queue is sized to store only a small number ofnotifications, the bulk of queue storage being provided by thenotification area in the multi-function multiport memory bank. As longas space is available on-chip (on the multiport) to hold thenotifications for a queue, the notification area is completely bypassed.When on-chip space runs out, the notification area acts as the large“middle” of the queue, with a few notifications at the head and tailbeing held on-chip.

While the size of the notification area will tend to limit the numbersof dropped packets, occasionally a queue will become full. Outputrequest processor includes a drop engine (not shown) for determiningwhich entries in a particular queue are to be dropped based on apredefined algorithm. In one embodiment, the drop engine institutes aprogrammable random early drop routine. The routine is programmable inthat the user can define one or more parameters, random in that a randomnumber generator is used to determine whether a entry will be dropped.Early refers dropping from the head of the queue.

The programmable random early drop routine may be implemented insoftware and when executed performs the following sequence ofoperations. The process begins by calculating the amount of data storedin a particular queue. This information is stored in the form of afraction (or percentage) of fullness. Thereafter, a drop criterion isdetermined based on the fraction of fullness. In one embodiment, a tableof drop criterion values ranging from zero to one is mapped againstfullness fractional values. The drop engine then derives a random numberfrom zero to one. The random number may be generated by a random numbergenerator or other means as is known in the art. A comparison is madebetween the random number generated and the drop criterion value.Thereafter, the entry at the head of the particular queue is dropped ifthe random number generated is larger than the drop criterion.Alternatively, the drop engine could avoid the drop if the random numbergenerated is less than the drop criterion. The drop engine operates oneach queue at a regular interval to assure that the queues do notoverflow and a orderly method of dropping packets is achieved ifrequired. This process is extremely helpful when transmitting packetsacross the Internet.

b) Per Bank Notification Queues

Each stream includes four queues 1700 that are serviced using a weightedround robin discipline. The weighting is used to reflect the priorityassociated with a given queue. For example, the four queues for a givenstream may be serviced in the following ratios: Q1 at 50%, Q2 at 25%, Q3at 15% and Q4 at 10%.

The multi-function multiport maintains four cell pointers for eachqueue: start, end, head, and tail. In one embodiment, each pointer is 23bits long and can address the entire memory associated with themulti-function multiport. The start and end pointers mark the boundariesof the queue's region, while the head and tail pointers point to thenext cell (notification) to read and next cell o write respectively. Thehead and tail pointers are restricted to align within the region definedby the start and end pointers, and standard wraparound arithmetic isperformed when incrementing these pointers.

Given the description above, it should be clear that the region for aqueue can be as small as one cell and as large as the entire memorybank. It is up to the software to configure the pointers atinitialization time to define the sizes of the regions, and to ensurethat regions are non-overlapping with each other and with the memoryallocated to the global packet buffer.

Typically, the software is used to allocate memory to a streamproportional to the stream's bandwidth.

C) Read Request Generation

Output request processor 306 services the queues to extractnotifications from the head regions of queues 1700. Output requestprocessor generates a first read request based on the full addressreceived from output switch 102. Thereafter subsequent read requests aregenerated for transmission to the input switch based on the offsetinformation provided in the request (from cell 1600) or indirect cells(as will be described below). Read requests include a stream identifierand a full address. Read requests are sent by the output requestprocessor to an appropriate read request queue 305. One read requestqueue 305 is provided for each bank of memory.

If the packet length, as determined from the route information providedwith the cell 1600, is greater than five (5) cells, then the outputrequest processor first requests the transfer (read from memory) of thefirst indirect cell associated with the packet. This is accomplished bycomputing the address of the indirect cell based on the full address andthe offsets provided in cell 1600. After the indirect cell request isgenerated, the output request processor generates read requests for theremaining cells in the packet based on the full address and the offsetsprovided in cell 1600. Upon receipt of a indirect cell from the outputswitch 102, output request processor continues to generate read requestsfor the remaining cells in the packet based on the offset informationcontained within the indirect cell.

Subsequent indirect cells are retrieved in a similar fashion. That is,at the time for reading the next indirect cell, the address of the nextindirect cell is computed based on the last offset stored in theprevious indirect cell. The timing of retrieving the indirect cells isaccomplished such that no delays in the output stream are incurred. Eachsubsequent indirect cell is retrieved prior to the end of the processingof the prior indirect cell. In this way, once the output stream isinitialized, no buffering of data is required and no interruptions dueto the latency associated with the retrieval process are experienced.

Output requests to an individual memory bank are processed strictly inorder. That is, the multi-function multiport may track each requestissued to a memory bank (through the read request queues) and is assuredthat the data received in response to a series of requests to the samememory bank will be strictly delivered according to the sequence orpattern in which they were issued. Output request processor 306 keepstrack of requests generated for each memory bank through the use ofreply queues (not shown). The request queue contains a stream number anda read address. When a request is issued to memory, the entry is removedfrom the request queue and the stream number portion is placed in anassociated reply queue. When a reply is received, the entry at the headof the reply queue is removed and the reply is sent to the stream number(in stream output buffer 312) indicated by the stream number retrievedfrom the reply queue.

As cells are received back at the multi-function multiport 150(responsive to the read requests), they are stored in an associatedstream output buffer 312. Stream output buffer 312 includes a pluralityof FIFOs, one for each stream. Each cell received for a stream is placedin the streams associated FIFO. For given packet, the multi-functionmultiport stores a fixed number of cells (in the FIFO) required toprovide a streamed output prior to initializing the output of the streamto line output interface 308. In one embodiment of the presentinvention, twelve cells are stored prior to beginning output (streamdata) from the output port. The selection of the number of cells forstorage in output buffer 312 is based on the latency in the read process(number of clock cycles between a read request from an multi-functionmultiport and the arrival of the cell associated with the read requestto the output port).

Output formatter 314 receives the cells from output buffer 312 andcouples the data with media header information stored in memory 310.Each request (notification) received from output switch 102 includes anext hop index. The next hop index indicates the starting address inmemory 310 of the media header information associated with a given typeof transmission (derived from the destination of the packet). Mediaheader information stored in memory 310 may be loaded uponinitialization of the router and updated by the controller as required.Output formatter 314 couples the cell data returned from memory with theappropriate media header to generate a proper packet for transfer out ofrouter 20 on the line output interface 308.

Packet Routing Overview

Referring now to FIG. 18, in a method of routing packets through aswitch a packet is received at a multi-function multiport (1800). Themulti-function multiport divides the packet into fixed length cells andtransfers the cells to an input switch (1802). Input switch removes thekey information from the first cell in a packet and stores ittemporarily in a key buffer (1804). Thereafter the input switch routesthe cells to memory banks resident in the multi-function multiports in atime division multiplexed manner (1806). The input switch stores thefirst address in memory where the first cell is stored and computesoffsets for each additional cell associated with the offset in memoryfor the next contiguous memory bank into which the next cell is written(1808). The input switch creates indirect cells to store linkinginformation for the packet if the packet length exceeds five cells(1810). If the number of cells exceeds the number of available offsetsin an indirect cell, then the old indirect cell is stored in memory anda new indirect cell is created and loaded based on the offsetscalculated for each new cell received at the input switch.

When the packet (and its indirect cells if any) have been stored inmemory, then the key, full address of the first cell and offsetinformation is transferred as a look-up request to a controller (1814).The controller performs a best match look-up and generates a result ofthe look-up. The result includes the destination port (multi-functionmultiport), address, offset information and next hop index (1816). Anotification including the result is transferred to the output switchfor transfer to the appropriate multi-function multiport (1818).

Upon receipt of a notification, the multi-function multiport generatesread requests a cell at a time to the input switch for the dataassociated with the packet (1820). The input switch issues the readrequests in a time division multiplexed fashion generating a singlerequest to each memory bank per cell slot (1822). When the memory bankreceives the request from the input switch, cell data and amulti-function multiport identifier associated with the request aretransferred to the output switch (1824). Again, at each cell slot, theoutput switch transfers a single cell to each of the multi-functionmultiports. Upon receipt, the multi-function multiport couples the celldata with media header information and streams the data to thedestination (1826).

System-Level View

Up to S streams, where S=s*b+1, may be in process in the systemconcurrently for b active banks (+1 is for the stream corresponding tothe control channel (the control channel stream is used to controlconfiguration and operation of elements in the router). Each streamconsists of a sequence of packets of arbitrary length L with arbitraryinter-packet gaps. In one embodiment L is less than 64 KBytes. A streamoperates at a constant rate R Bytes/sec when it is receiving a packet,and a rate of 0 Bytes/sec when it is not. Thus, there are no “holes”during the reception of a packet. The value R for a stream is determinedat system initialization time and Rmin and Rmax are system constantsdefined at design time. In one embodiment, Rmin is approximately 4MBytes/sec and Rmax is approximately 320 MBytes/sec.

As described earlier, each packet is divided up into 64-byte cellsbefore being written to memory. If all packets are long compared to 64bytes, then the rate of cells generated by a stream is simply R/64 or 0depending on whether a packet is being received or not. Unfortunately,because real life packets do not satisfy this assumption, theinstantaneous rate of cell generation depends on the lengths of thepackets being received, and this rate will typically be larger than R/64because of unoccupied bytes in cells. Because of this dependency, themost we can say is that a stream is either generating cells at some ratebetween R/64 and a small factor f times this rate, or it is not. Theworst-case value of f for IP is 128/65, which occurs for 65 Byte packetswhere one cell is full and the other carries a single byte. Forrealistic network traffic, however, the average value of f should bearound 1.2, with bursts of 40 Byte packets causing f to go as high as1.6 for short periods.

Let the cells generated by a given stream be numbered I, I+1, I+2, . . .etc. As was described above, cells are written to sequentiallyincreasing bank numbers I mod b. The virtual address to which aparticular cell is written is determined largely by the cell's arrivaltime into the system. For each active memory bank, the system maintainsa write pointer that determines where the next cell directed to thatbank will be written. The pointers are kept in the input switch, and sois the logic to switch incoming cells to the appropriate bank. When theinput switch receives a cell, it knows which bank the cell should bewritten to by the cell's ordinal number. The input switch looks up thewrite pointer for the destination bank, writes the cell, and thenincrements the bank pointer to point to the next location.

At system initialization, all bank pointers in the input switch are setto 0, and the S streams in the system are initialized with starting banknumbers such that approximately the same number of streams are pointedto each of the b active banks. Although the starting bank numbers aredistributed evenly, they do not have to be. Distributing them evenlysimply makes the magnitude of the maximum negative offset a littlesmaller. FIG. 19 shows the starting configuration for a system with b=8.

Over time, the bank pointers move as cells are written to the buffer,but they move in a compact group because each stream writes its packetsto sequentially increasing bank numbers mod b. The overall speed of thegroup is determined by the sums of the instantaneous bandwidths on allthe streams, and can vary from 0 cells/sec when no packets are beingreceived to a maximum of 400/64=6.25M cells/sec when all interfaces arerunning at peak capacity.

ALTERNATIVE EMBODIMENTS

In an alternative embodiment, multi-function multiport 150 may bedivided into separate functional blocks and embodied in separatephysical structures. For example, each multi-function multiport may beseparated into an input port, an output port and a memory. Memory mayinclude both the global data buffer and the notification queues.Alternatively, memory may be divided further with the notificationqueues physically manifested as a part of an output port.

The present invention has been described in terms of specificembodiments, which are illustrative of the invention and not to beconstrued as limiting. Other embodiments are within the scope of thefollowing claims.

What is claimed is:
 1. A router for switching a data packet from asource to a destination in a network, the router comprising: an inputport for receiving the data packet, the data packet formatted inaccordance with a first protocol; a divider for dividing the data packetinto uniform length cells for temporary storage in the router, thedivider dividing the data packet without changing the formatting of thedata packet; a distributed memory comprising a plurality of physicallyseparated memory banks, each memory bank including a global data areafor storing a portion of the data packet; an input switch for streamingacross the memory banks uniform portions of the data packet such thatconsecutive cells associated with the data packet are stored inconsecutive banks of the distributed memory; a controller fordetermining packet routing through the router; an output switch forextracting in order the portions of the data packet stored in the globaldata area of each memory bank including reassembling the cells in properorder to reconstruct the data packet formatted in accordance with thefirst protocol and forwarding the data packet to an appropriate outputport; and an output port for transferring the data packet to thedestination.
 2. An apparatus for use in a router, the router forswitching data packets from a source to a destination in a network, therouter including one or more input ports for receiving data packets, adivider for dividing the data packets into uniform portions withoutchanging the format of the data packet and one or more output portsoperable to forward the data packets to other devices on the networkbased on routing decisions made by the router, the apparatus comprising:a distributed memory comprising a plurality of physically separatedmemory banks; a switch coupled to the divider operable to stream theuniform portions to the memory banks where consecutive cells associatedwith a data packet are stored in consecutive memory banks; each memorybank operable to store uniform portions of a data packet received fromthe source and linking information to allow for the extraction of theuniform portions of the data packet from the plurality of memory banksin proper order after a routing determination has been made by therout4er such that the data packet as originally formatted can be routedto another device in the network.
 3. The apparatus of claim 2 whereinthe distributed memory includes an output queue for storing anotification indicative of the routing of the data packet through therouter.
 4. The apparatus of claim 3 wherein the notification includeslinking information for retrieving at least a first cell of the datapacket from the distributed memory.
 5. The apparatus of claim 4 whereinthe notification includes linking information for the first 5 cells ofthe data packet.
 6. The apparatus of claim 3 wherein the notificationincludes an address for an indirect cell, the indirect cell stored inthe distributed memory and including linking information for extractingcells in order from the distributed memory.
 7. The apparatus of claim 3wherein each memory bank includes a global data area for storingportions of data packets and a notification area for storingnotifications.
 8. The apparatus of claim 7 wherein the notification areais sized to be ⅕ of a size of the global data area for a given memorybank.
 9. The apparatus of claim 8 further including a plurality ofmulti-function multiports, each multi-function multiport including oneor more input ports and output ports for receiving and transmitting datapackets through the router.
 10. The apparatus of claim 9 wherein aportion of the distributed memory is located within each multi-functionmultiport such that each multi-function multiport includes a memory bankhaving a global data area and a notification area.
 11. The apparatus ofclaim 10 wherein the notification area of a given multi-functionmultiport stores notifications for data packets to be routed through anoutput port of the given multi-function multiport.
 12. The apparatus ofclaim 2 wherein memory reads and writes to and from the distributedmemory are sized to be 64 bytes.
 13. The apparatus of claim 2 furtherincluding mapping means for mapping from a virtual address space to aphysical address space associated with the distributed memory, themapping means for detecting aged packets in memory and allowing for easyoverwriting thereof such that garbage collection of aged packets is notrequired.
 14. A method of routing a data packet through a router in asystem transmitting data packets between a source and a destination overa network including the router, the method comprising: receiving thedata packet; dividing the data packet into cells of a fixed size;storing the cells in a distributed memory, the distributed memoryincluding two or more memory banks where consecutive cells from the datapacket are stored in consecutive banks of the distributed memory; andstoring linking information in one bank of the memory for linking cellsof the data packet that are stored throughout the distributed memory,the linking information used for extracting the cells in order fortransmission from the router to the destination.
 15. A method of storinga data packet in a router while a look-up engine determines a properpath through the router for the data packet in a system transmittingdata packets between a source and a destination over a network includingthe router, the method comprising: dividing the data packet into fixedlength cells without changing the formatting of the data packet; storingthe cells and linking information for reconstructing the data packetacross a global data buffer, the global data buffer including two ormore memory banks where the data packet is divided among the memorybanks including storing consecutive cells associated with a data packetin consecutive memory banks of the global data buffer; and a switch forgathering cells associated with the data packet from the global databuffer and reconstructing the data packet prior to forwarding the packetto the destination.
 16. A method for determining which data packetsstored in a router are to be passed through the router in a systemtransmitting data packets between a source and a destination over anetwork including the router, the method comprising: determining a routethrough the router for a data packet, the route including a notificationindicating a starting address in memory where the data packet is stored;processing the notification including queuing the notification withother notifications to be output on the same output port of the routerin a stream queue; determining the fullness of the stream queue;determining a drop criterion based in part on fullness of the streamqueue; generating a random number; comparing the random number and thedrop criterion and dropping the notification based upon results of thecomparison.
 17. The method of claim 16 wherein the step of determiningthe fullness of the stream queue is performed when the data packetreaches a head of the stream queue.
 18. The method of claim 16 whereinthe fullness is a percentage and the random number generated is betweenzero and one.